Systems and methods for prioritized rendering and streaming based on risk maps that predict change in a three-dimensional environment

ABSTRACT

A system prioritizes the rendering and streaming of image data based on risk maps that predict change in a three-dimensional (“3D”) environment. The system receives primitives that are distributed across a 3D space to represent the 3D environment. The system generates a first image based on primitives that fall within a first view frustum, and generates a risk map with a risk value for each particular pixel of the first image. Each risk value quantifies a probability that a pixel of the first image associated with that risk value changes as a result of changing the first view frustum to a second view frustum. The system then performs an out-of-order rendering of primitives that fall within the second view frustum based on the risk value for each first image pixel that is replaced in a second image with a rendered primitive from the second view frustum.

CLAIM OF BENEFIT TO RELATED APPLICATIONS

This application is a continuation of U.S. nonprovisional applicationSer. No. 18/048,973 entitled “Systems and Methods for PrioritizedRendering and Streaming Based on Risk Maps that Predict Change in aThree-Dimensional Environment”, filed Oct. 24, 2022. The contents ofapplication Ser. No. 18/048,973 are hereby incorporated by reference.

BACKGROUND

A point cloud is a high resolution encoding a three-dimensional (“3D”)environment. The point cloud has millions or billions of data pointsthat represent different scanned points or surfaces within the 3Denvironment. The resources required to render such a large number ofdata points for any scene within the 3D environment, or to rendermovement or different scenes within the 3D environment at a smooth ratemay exceed the resources of user devices used to view and/or interactwith the 3D environment.

Accordingly, the point cloud processing and/or rendering may beoffloaded from the user devices to a remote resource pool. The remoteresource pool renders the large volume of data points for differentscenes of an animation and streams the rendered data to a user device.Even in this scenario, the amount of rendered data that is generated foreach frame or scene of an animation or movement within the 3Denvironment may be too large to transmit over existing data networks.For instance, the time it takes to transmit and present the rendereddata for different scenes on the user device may be greater than therate at which those scenes are rendered and/or are to be displayed onthe user device.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates performing an out-of-order rendering and streaming ofimage data based on a risk map in accordance with some embodimentspresented herein.

FIG. 2 illustrates an example mapping of a risk map to point cloud datapoints in accordance with some embodiments presented herein.

FIG. 3 presents a process for the predictive rendering and streaming ofimage data for a next scene based on risks computed for pixels of acurrent scene in accordance with some embodiments presented herein.

FIG. 4 illustrates an example of the prioritized selection and renderingof the data points within a view frustum of a current scene based on arisk map that is generated for the pixels of a previous scene inaccordance with some embodiments presented herein.

FIG. 5 presents a process for generating a risk map in accordance withsome embodiments presented herein.

FIG. 6 illustrates an example of assessing the risk associated with aparticular pixel in the risk map in accordance with some embodimentspresented herein.

FIG. 7 illustrates example components of one or more devices, accordingto one or more embodiments described herein.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The following detailed description refers to the accompanying drawings.The same reference numbers in different drawings may identify the sameor similar elements.

Provided are systems and methods for prioritized rendering and streamingof image data based on risk maps that predict change in athree-dimensional (“3D”) environment. Specifically, the systems andmethods include a rendering system that uses the risk maps to prioritizethe rendering of different parts of a selected scene within the 3Denvironment, and/or to prioritize the sending of rendered image data forthe different parts of the scene. For instance, the rendering systemuses the risk maps to determine which pixels from a currently renderedscene are most likely to change when there is movement to a next scene,prioritizes the rendering of the image data or data points for thosepixels in the next scene over the rendering of image data for otherpixels of the next scene, and/or prioritizes the sending of data packetsencoding the rendered pixels for the next scene by sending the datapackets for the pixels that are most likely to change before the datapackets for other pixels of the next scene.

In some embodiments, the rendering system generates the risk maps usinglinear gradients. The linear gradients quantify the amount of changeoccurring around or some specified distance from each rendered datapoint of the current scene. A risk assessment is associated with thepixels of the current scene based on the quantified amount of change forthe image data or data points that produced those pixels. The riskassessment predicts the likelihood that a pixel will change as therepresented scene from the 3D environment changes due to movement withinthe 3D environment.

FIG. 1 illustrates performing an out-of-order rendering and streaming ofimage data based on a risk map in accordance with some embodimentspresented herein. Rendering system 100 generates (at 102) current scene101 by rendering a set of image of data from a 3D environment that iswithin a current view frustum.

Rendering system 100 generates (at 104) risk map 103 for current scene101. Risk map 103 associates probability values to different parts ofcurrent scene 101 that quantify the likelihood that those differentparts of current scene 101 change for a next scene of the 3Denvironment. In some embodiments, the different probability values ofrisk map 103 are computed for individual pixels or sets of pixels ofcurrent scene 101. Each set of pixels with a computed risk probabilitymay include a part of current scene 101 that has little or no varianceamongst the pixels representing that part of current scene 101. Forinstance, a single probability value may be associated with a set ofpixels that represent the same uniform surface or object in currentscene 101.

Rendering system 100 provides (at 106) current scene 101 to one or moretarget devices 105. Target devices 105 may include devices with displaysthat have insufficient resources to render the different scenes of the3D environment at a sufficient frame rate and/or at a desiredresolution, and therefore rely on rendering system 100 to render thedifferent scenes of the 3D environment on their behalf.

Rendering system 100 performs (at 108) the out-of-order rendering of anext scene in response to a change of the view frustum. Performing (at108) the out-of-order rendering includes rendering the parts of the nextscene that align with or correspond to the parts of risk map 103 withthe largest probability values before rendering other parts of the nextscene. Accordingly, rendering system 100 renders the data points in thechanged view frustum for the next scene in a non-sequential order.

Rendering system 100 performs (at 110) the out-of-order streaming bytransmitting the rendered image data for the parts of the next scenethat align with or map to the parts of risk map 103 with the largestprobabilities before transmitting the rendered image data for otherparts of the next scene to one or more target devices 105. In performing(at 110) the out-of-order streaming, rendering system 100 ensures thatone or more target device 105 receive and/or present the parts of thenext scene that are associated with the greatest likelihood of change sothat if there is insufficient bandwidth to stream all the rendered imagedata for the next scene, target devices 105 nevertheless update theparts of current scene 101 where there is the greatest expected amountof visual differentiation first.

FIG. 2 illustrates an example mapping of a risk map to point cloud datapoints in accordance with some embodiments presented herein. Renderingsystem 100 receives (at 202) 3D data representing a 3D environment. Insome embodiments, the 3D data includes data points of a point cloud. Insome other embodiments, the 3D data includes meshes, polygons, and/orother constructs for creating the 3D environment.

The data points of a point cloud differ from pixels of a two-dimensional(“2D”) image, because certain regions of the point cloud have no datapoints, lower densities of data points, and/or higher densities of datapoints based on varying amounts of visual information that is detectedor scanned at those regions. Additionally, the position of the pointcloud data points are defined in 3D space using x, y, and z coordinatevalues. In contrast, pixels of a 2D image are defined in 2D space andhave a uniform density or fixed arrangement that is defined by theresolution of the 2D image. In other words, the point cloud data pointshave a non-uniform placement or positioning, whereas the 2D image haspixel data for each pixel of a defined resolution (e.g., 640×480,800×600, etc.).

Each point cloud data point is defined with a plurality of elements. Theplurality of elements includes a first set of positional elements, and asecond set of non-positional or descriptive elements.

The positional elements include coordinates within a 3D space. Forinstance, each point cloud data point includes x-coordinate,y-coordinate, and z-coordinate elements to capture the position of acorresponding physical point from a surface, feature, or object. Thepositional elements further include a surface normal. The surface normaldefines the angle, direction, or orientation that the surface, feature,or object represented by the data point faces or is exposed to. Morespecifically, the surface normal is a line, ray, or vector that isperpendicular to the scanned surface, feature, or object represented bythe data point.

The non-positional elements may include information about the detectedcharacteristics of the surface, feature, or object at a correspondingposition in the scanned scene. The characteristics may correspond to adetected color. The color may be represented using red, green, blue(“RGB”), and/or other values. In some embodiments, a data point may havemultiple sets of non-positional elements with each set of non-positionalelements storing intensity values or other hyperspectral values detectedacross a different band of the electromagnetic spectrum. For instance, afirst set of non-positional elements may store values measured in the800 to 2,500 nanometer wavelengths for near-infrared light, and a secondset of non-positional elements may store values measured in the 10 to400 nanometer wavelengths for ultraviolet light from the position of thereal-world object identified by the associated positional elements ofthe data point.

In some embodiments, the non-positional element store other measured orderived characteristics including the chrominance, hardness,translucence, reflectivity, luminance, metallic characteristics,roughness, specular, diffuse, albedo, index of refraction (“IOR”),and/or other properties from the imaged surface, feature, or object. Insome embodiments, the non-positional elements may directly identify amaterial property or other classification for a data point. Forinstance, a first data point may be defined with a non-positionalelement with a value that identifies the material property of“aluminum”, a second data point may be defined with a non-positionalelement with a value that identifies the material property of “steel”,and a third data point may be defined with a non-positional element witha value that identifies the material property of “plastic”.

In some embodiments, the characteristics may be related to properties ofthe device used to generate each of the data points. For instance, thecharacteristics may include a Tesla strength value to quantify thestrength of the magnetic field that was used in detecting and/or imagingthe surface, feature, or object represented by a particular point clouddata point. In some embodiments, the non-positional elements may includeenergy, audio or sound, and/or other characteristics of the device orthe object being imaged. Accordingly, the non-positional elements caninclude any property of the imaged surface, feature, or article (e.g.,hue, saturation, brightness, reflectivity, etc.) or of the device usedto capture the object part at a corresponding data point in 3D space.

Each point cloud data point may include an array of elements. The arrayof elements may provide the positioning of the data point in 3D space aswell as one or more characteristics of that data point. For instance, apoint cloud data point may be stored and/or represented as an array ofelements with some combination of x-coordinate, y-coordinate,z-coordinate, RGB values, values measured in the near-infrared band,values measured in the far-infrared band, values measured in theultraviolet band, values measured in other hyperspectral bands,chrominance, hardness, translucence, reflectivity, luminance, metalliccharacteristics, roughness, specular, diffuse, albedo, IOR, tesla,and/or other values.

The same positional and non-positional elements apply to other 3Dformats that encode a 3D environment with other primitives such asmeshes or polygons. The meshes or polygons of other 3D formats may alsobe non-uniformly distributed across a 3D space. The term data point asused hereafter therefore interchangeably refers to data points of apoint cloud or meshes, polygons, and/or primitives or constructs ofother 3D formats that used to represent a 3D environment.

Rendering system 100 configures (at 204) a view frustum from which torender a first scene of the 3D environment. The view frustum may beconfigured by a user. For instance, the user may position a virtualcamera to set the field-of-view from which to visualize the first scene.Alternatively, the view frustum may be preconfigured in the received (at202) 3D data, and may correspond to an initial or default field-of-viewfrom which to present the 3D environment.

Rendering system 100 selects (at 206) the data points that arepositioned within the view frustum, and generates (at 208) first scene201 based on the positional and non-positional elements of the selected(at 206) data points. Generating (at 208) first scene 201 includesrendering the selected (at 206) data points in the 3D space encompassedby the current view frustum to create the pixels that provide a 2Dvisualization of the 3D environment from the current view frustum.Accordingly, rendering system 100 converts the 3D image data (e.g., datapoints, polygons, meshes, etc.) within the current view frustum into 2Dpixels of first scene 201.

Rendering system 100 generates (at 210) risk map 203 to associate a riskwith each pixel of first scene 201. The risk associated with aparticular pixel quantifies the probability that the attributes of theparticular pixel will change in a next scene or as the current viewfrustum changes or shifts away from a current position and/ororientation. The pixel attributes include the pixel color components,such as RGB values, and/or other values (e.g., gamma, transparency,etc.) used by a display to present the visual characteristics of thatpixel.

Rendering system 100 uses risk map 203 to prioritize the order withwhich pixels of a next scene are rendered and/or streamed to one or moretarget devices 105. Specifically, rendering system 100 tracks (at 210)the changes to the view frustum. The view frustum changes define thenext scene from the 3D environment that is to be rendered and/orpresented. The view frustum changes are gradual movements from a lastposition or orientation to an adjusted position or orientation, ratherthan an abrupt movement of the view frustum from one location in the 3Denvironment to an entirely different location in the 3D environment.

Rendering system 100 selects (at 212) the data points that arepositioned within the adjusted view frustum for the next scene.Rendering system 100 maps (at 214) risk map 203 onto the selected (at212) data points of the next scene. The mapping (at 214) identifieswhich data points of the selected (at 212) of the next scene contain theimage data for rendering the pixels of risk map 203 with the highestlikelihood of changing from the current scene to the next scene. In someembodiments, the mapping (at 214) involves aligning risk map 203 withthe adjusted view frustum, projecting rays from the render position ofthe adjusted view frustum through each pixel of risk map 203, andidentifying the data points that intersect or contact the ray projectedthrough the pixels associated with the greatest risk of changing.Rendering system 100 then prioritizes the rendering of the selected (at212) data points for the next scene in order according to the datapoints that intersect or contact the rays projected through the pixelsassociated with the greatest risk to the lowest risk, and prioritizesthe streaming of the rendered image data accordingly.

FIG. 3 presents a process 300 for the predictive rendering and streamingof image data for a next scene based on risks computed for pixels of acurrent scene in accordance with some embodiments presented herein.Process 300 is implemented by rendering system 100. Rendering system 100includes one or more devices that accelerate the processing and/orrendering of complex 3D environments for different target devices. Theone or more devices include one or more hardware processors (e.g.,central processing units, graphics processing units, acceleratedprocessing units, etc.), memory, storage, network controllers, and/orother compute resources for the risk map generating, the 3D image datarendering, the predictive streaming of image data, and/or otheroperations related to the out-of-order rendering and presentation ofdifferent scenes from a 3D environment. In some embodiments, renderingsystem 100 may be hosted in the cloud or may be remote to the targetdevices that request the different scenes of a 3D environment fromrendering system 100 over a data network, and that receive the renderedimage data for the requested scenes from rendering system 100 in theprioritized out-of-order sequencing determined from the generated riskmaps.

Process 300 includes generating (at 302) a first image for a part of a3D environment that is within a view frustum. Generating the first imageincludes rendering the data points, polygons, meshes, and/or otherprimitives that represent that part of the 3D environment in a pointcloud or other 3D encoding of the 3D environment.

Process 300 includes generating (at 304) a risk map with a riskassessment for each pixel or set of pixels in the first image. In someembodiments, rendering system 100 generates (at 304) the risk map usinglinear gradients. The linear gradient that defines the risk associatedwith a particular pixel in the risk map may quantify the amount ofchange surrounding or neighboring the one or more data points thatproduced that particular pixel in the first image. In some embodiments,the linear gradient, that quantifies the risk associated with a pixelthat is rendered from a particular data point, tracks the transition incolor values and/or other descriptive characteristics between theparticular data point and its neighboring data points or in a volume of3D space surrounding the particular data point in the represented 3Denvironment. In some embodiments, the linear gradient tracks the numberof other data points or different objects within a volume of 3D spacesurrounding the particular data point in the represented 3D environment.In any case, the risk assessment associated with a particular pixel inthe first image is derived based on the amount of change occurring inthe region surrounding the one or more data points that were rendered inorder to produce that particular pixel.

Process 300 includes receiving (at 306) input that changes the viewfrustum. The input involves moving, rotating, and/or otherwise changingthe view frustum from a first position and/or orientation, that resultedin the first image, to a second position and/or orientation. In otherwords, the input gradually moves the view frustum so that some of thedata points used to render the first image remain within the viewfrustum at the second position and/or orientation.

Process 300 includes mapping (at 308) the risk map created for thepixels of the first image onto the adjusted view frustum. In someembodiments, rendering system 100 overlays or otherwise aligns the riskmap with the field-of-view or image plane of the adjusted view frustumin the 3D environment. The image plane corresponds to the 2D flatteningof the data points or the part of the 3D environment that falls withinadjusted view frustum.

Process 300 includes determining (at 310) the subset of data pointswithin the adjusted view frustum that contain the image data forrendering the pixels of a second image that correspond to pixels in therisk map with the highest risk of changing and/or the greatest risk thathave yet to be rendered as part of the second image. Determining (at310) the subset of data points includes projecting rays from the virtualcamera or render position, that defines the adjusted view frustum,through the risk map that is aligned over the adjusted view frustum, andselecting the subset of data points that intersect or make contact withthe rays projected through the pixels of the risk map that areassociated with the highest risk.

Process 300 includes rendering (at 312) the subset of data points toproduce a visualization for the pixels of the second image thatcorrespond to the pixels in the risk map with the highest probability ofchange or the greatest risk. Accordingly, rendering system 100 performsan out-of-order rendering of the second image that provides a 2Dvisualization of the 3D environment from the adjusted view frustum.Specifically, rendering system 100 generates the pixels of the secondimage in a non-sequential order determined according to the riskassociated with different pixels of the risk map.

Process 300 includes streaming (at 314) the pixels that are generatedfrom rendering (at 312) the subset of data points or image data to oneor more target devices. Accordingly, rendering system 100 sends thegenerated pixels of the second image that correspond to the pixels inthe risk map with the greatest risk before sending other pixels of thesecond image to the target devices.

Process 300 includes determining (at 316) whether the adjusted viewfrustum is changed to a different part of the 3D environment before alldata points within the adjusted view frustum for all pixels of thesecond image have been rendered and/or streamed. In response todetermining (at 316—No) that the adjusted view frustum has not changed,process 300 determines (at 310) the next subset of data points for theremaining pixels of the second image with the greatest risk, renders (at312) that next subset of data points, and streams (at 314) the renderedresults to the target devices. In response to determining (at 316—Yes)that the adjusted view frustum has changed, process 300 halts theout-of-order prioritized rendering and streaming of the second imagepixels that produce a visualization of the adjusted view frustum, andperforms (at 318) the out-of-order prioritized rendering and streamingof pixels for a third image that collectively produce a visualization ofthe 3D environment from the new position and/or orientation of the viewfrustum. Performing (at 318) the out-of-order prioritized rendering andstreaming includes updating the risk map by computing the risk that isassociated with pixels of last generated scene, selecting data pointswithin the updated view frustum based on the updated risk map, andrendering and streaming the data points in the order that they areselected based on the updated risk map.

FIG. 4 illustrates an example of the prioritized selection and renderingof the data points within a view frustum of a current scene based on arisk map that is generated for the pixels of a previous scene inaccordance with some embodiments presented herein. Rendering system 100generates (at 402) risk map 401 to assess the risk associated with thepixels of the previous scene changing if the view frustum is to change.For simplicity, risk map 401 is illustrated with four pixels or regionswith an assessed risk. In actually, risk map 401 will have the sameresolution or number of pixels as the image generated for the previousscene and/or all images or scenes at which the 3D environment isvisualized, and a risk assessment for each pixel or a set of adjacentpixels.

Rendering system 100 tracks (at 404) movement of the view frustum from aprevious position that captured the previous scene to a current positionthat captures the current scene. Tracking (at 404) the movement of theview frustum includes tracking the position and/or orientation of avirtual camera or a render position from which the view frustumoriginates and/or is defined. The size or resolution of the view frustumdoes not change as the virtual camera or render position is changed.

Rendering system 100 positions (at 406) risk map 401 onto the adjustedview frustum at the current position. Rendering system 100 projects (at408) lines, vectors, or rays from the virtual camera or render positionin the angle of and/or through the overlaid pixels of risk map 401. Insome embodiments, rendering system 100 projects (at 408) the rays inorder through the pixels of risk map 401 that are associated with thegreatest risk to the pixels that are associated with the least risk.

Rendering system 100 detects which data points or other image data(e.g., polygons, meshes, and/or other 3D primitives) in the 3Denvironment intersect or are contacted by the projected (at 408) rays.Rendering system 100 selects the contacted data points, and renders (at410) those data points out-of-order according to the risk associatedwith the pixel from risk map 401 that the intersected ray passedthrough.

In some embodiments, rendering system 100 generates and/or representsthe risk map for the pixels of a generated scene using linear gradients.The linear gradients track the amount of change surrounding the datapoints that were rendered to produce different pixels of the risk map.

FIG. 5 presents a process 500 for generating a risk map in accordancewith some embodiments presented herein. Process 500 is implemented byrendering system 100.

Process 500 includes receiving (at 502) data points, meshes, polygons,and/or other primitives that collectively represent and produce avisualization of a 3D environment. For instance, rendering system 100receives (at 502) a point cloud that is 3D representation of a scanned3D environment.

Process 500 includes defining (at 504) a position and/or orientation fora virtual camera or a render position. In some embodiments, user inputdefines (at 504) the position and/or orientation for the virtual camera.In some other embodiments, an initial position and/or orientation forthe virtual camera is defined as part of the image data for the 3Denvironment, or is defined (at 504) by rendering system 100 or anapplication executed by rendering system 100.

Process 500 includes defining (at 506) the view frustum from theposition and/or orientation of the virtual camera and/or renderposition. Defining (at 506) the view frustum includes extending a cone,pyramid, or other form from the virtual camera position or renderposition some distance into the 3D environment to establish afield-of-view for a first visualization of the 3D environment from thevirtual camera position.

Process 500 includes projecting (at 508) a number of lines, rays, orvectors from the virtual camera or render position through the viewfrustum. In some embodiments, rendering system 100 projects a line forevery pixel that is to be generated for the scene or image captured bythe view frustum. Accordingly, the number of lines projected may beequal to the resolution at which the view frustum is rendered.

Process 500 includes detecting (at 510) a data point, mesh, polygon, orother primitive that is in the path of a projected line and/or thatintersects or contacts a projected line. Rendering system 100 detects(at 510) the data point that is first intersected by or contacted by theprojected line, wherein the data point that is first intersected by orcontacted by the projected line is the data point that is visible in therendered image of the view frustum and is not behind or obscured byother data points.

Process 500 includes computing (at 512) a linear gradient to assess therisk associated with the view frustum pixel that the line passedthrough. In some embodiments, computing (at 512) the linear gradientinvolves determining an amount of change surrounding or that is within aspecified distance of the contacted data point, mesh, polygon, or otherprimitive.

Rendering system 100 may determine the amount of change using thesurface normal of the contacted point. The surface normal may be encodedas a positional or non-positional element. The surface normal is avector that is perpendicular to the surface represented by the contacteddata point. In some embodiments, rendering system 100 assesses the riskassociated with the contacted data point based on the angle of thesurface normal relative to the projected line. In some such embodiments,a higher risk is associated with a smaller or more acute angle. Forinstance, the acute angle may indicate that the edge or end of theobject represented by the contacted data point is directed towards theview frustum such that a movement of the view frustum may shift awayfrom the object to another object producing a change in the pixelvisualization. Conversely, a large angle may indicate that a largerportion of the object represented by the contacted data point remainswithin the view frustum such that a movement of the view frustum willresult in the same object being presented within the view frustum,thereby reducing the risk of the pixel changing especially for a uniformobject. Even if the visual characteristics of the object change, thechanges are likely to be gradual which results in smaller changes to thepixels than if the view frustum transitions from one object to anentirely different object. The linear gradient may record the angle ofdifference between the surface normal and movement of the view frustumin different directions (e.g., up, down, left, right, etc.), therebyproviding a different risk assessment if the view frustum was moved indifferent directions.

In some embodiments, rendering system 100 reflects or bounces the lineoff the contacted data point, mesh, polygon, or other primitiveaccording to the surface normal, continues projecting the line a givendistance, and determines the number of reflections or data points thatthe line projects off of. A higher risk is associated with a largernumber of reflections or contacted data points because the larger numberof reelections or contacted data points indicates that there are othernearby objects that can change the visualization of the pixel if theview frustum is slightly moved off the first contacted data point. Thelinear gradient may record the number of reflections or objects detectedin different directions from the contacted data point.

In some embodiments, rendering system 100 generates a plane that isperpendicular to the surface normal of the contacted point or that isparallel with the alignment of neighboring data points. Rendering system100 determines the deviation in color and/or other descriptivecharacteristics between the data points in the generated plane, andassesses the risk associated with the contacted data point based on thedeviation. The linear gradient may record the different amounts ofdeviation if the view frustum was moved to different sides of thegenerated plane.

Process 500 includes generating (at 514) the risk map based on the riskassociated with the linear gradient that is computed (at 512) for eachpixel of the view frustum. The risk map provides a risk that quantifiesthe probability of the pixel changing as the view frustum changes. Insome embodiments, the risk map provides an overall risk associated witheach pixel, wherein the overall risk is determined from the lineargradient mapping of risk in different directions from the pixel. In someother embodiments, the risk map provides different risks associated withmoving the view frustum in different directions from each pixel.

FIG. 6 illustrates an example of assessing the risk associated with aparticular pixel in the risk map in accordance with some embodimentspresented herein. Rendering system 100 tracks (at 602) the virtualcamera position, and generates (at 604) an image with a set of pixelsbased on the data points, meshes, polygons, and/or other primitives fromthe 3D environment that fall within the view frustum of the currentvirtual camera position. For simplicity, FIG. 6 presents the image as a2×2 grid of pixels. Rendering system 100 projects (at 606) a ray fromthe virtual camera position through pixel 601 of the view frustum.

Rendering system 100 detects the intersection of the projected (at 606)ray and data point 603. Rendering system 100 generates (at 608) a planebased on the surface normal of the particular data point and/or thepositioning of data points adjacent to or that neighbor data point 603.Rendering system determines the amount of change surrounding data point603 across the generated plane, and computes (at 610) a risk score basedon the determined amount of change.

In some other embodiments, rendering system 100 reflects the projected(at 606) ray off data point 603 according to the surface normal of datapoint 603. Rendering system 100 determines the angle of the reflectedray and/or determines the amount of nearby change based on other datapoints that the reflected ray makes contact with and differences incolor values and/or other descriptive characteristics between thecontacted data points and data point 603. Rendering system 100 maycompute the risk score based on the determined angle or the determinedamount of nearby change.

FIG. 7 is a diagram of example components of device 700. Device 700 maybe used to implement one or more of the devices or systems describedabove (e.g., rendering system 100, scanners, imaging devices, etc.).Device 700 may include bus 710, processor 720, memory 730, inputcomponent 740, output component 750, and communication interface 760. Inanother implementation, device 700 may include additional, fewer,different, or differently arranged components.

Bus 710 may include one or more communication paths that permitcommunication among the components of device 700. Processor 720 mayinclude a processor, microprocessor, or processing logic that mayinterpret and execute instructions. Memory 730 may include any type ofdynamic storage device that may store information and instructions forexecution by processor 720, and/or any type of non-volatile storagedevice that may store information for use by processor 720.

Input component 740 may include a mechanism that permits an operator toinput information to device 700, such as a keyboard, a keypad, a button,a switch, etc. Output component 750 may include a mechanism that outputsinformation to the operator, such as a display, a speaker, one or moreLEDs, etc.

Communication interface 760 may include any transceiver-like mechanismthat enables device 700 to communicate with other devices and/orsystems. For example, communication interface 760 may include anEthernet interface, an optical interface, a coaxial interface, or thelike. Communication interface 760 may include a wireless communicationdevice, such as an infrared (“IR”) receiver, a Bluetooth® radio, or thelike. The wireless communication device may be coupled to an externaldevice, such as a remote control, a wireless keyboard, a mobiletelephone, etc. In some embodiments, device 700 may include more thanone communication interface 760. For instance, device 700 may include anoptical interface and an Ethernet interface.

Device 700 may perform certain operations relating to one or moreprocesses described above. Device 700 may perform these operations inresponse to processor 720 executing software instructions stored in acomputer-readable medium, such as memory 730. A computer-readable mediummay be defined as a non-transitory memory device. A memory device mayinclude space within a single physical memory device or spread acrossmultiple physical memory devices. The software instructions may be readinto memory 730 from another computer-readable medium or from anotherdevice. The software instructions stored in memory 730 may causeprocessor 720 to perform processes described herein. Alternatively,hardwired circuitry may be used in place of or in combination withsoftware instructions to implement processes described herein. Thus,implementations described herein are not limited to any specificcombination of hardware circuitry and software.

The foregoing description of implementations provides illustration anddescription, but is not intended to be exhaustive or to limit thepossible implementations to the precise form disclosed. Modificationsand variations are possible in light of the above disclosure or may beacquired from practice of the implementations.

The actual software code or specialized control hardware used toimplement an embodiment is not limiting of the embodiment. Thus, theoperation and behavior of the embodiment has been described withoutreference to the specific software code, it being understood thatsoftware and control hardware may be designed based on the descriptionherein.

For example, while series of messages, blocks, and/or signals have beendescribed with regard to some of the above figures, the order of themessages, blocks, and/or signals may be modified in otherimplementations. Further, non-dependent blocks and/or signals may beperformed in parallel. Additionally, while the figures have beendescribed in the context of particular devices performing particularacts, in practice, one or more other devices may perform some or all ofthese acts in lieu of, or in addition to, the above-mentioned devices.

Even though particular combinations of features are recited in theclaims and/or disclosed in the specification, these combinations are notintended to limit the disclosure of the possible implementations. Infact, many of these features may be combined in ways not specificallyrecited in the claims and/or disclosed in the specification. Althougheach dependent claim listed below may directly depend on only one otherclaim, the disclosure of the possible implementations includes eachdependent claim in combination with every other claim in the claim set.

Further, while certain connections or devices are shown, in practice,additional, fewer, or different, connections or devices may be used.Furthermore, while various devices and networks are shown separately, inpractice, the functionality of multiple devices may be performed by asingle device, or the functionality of one device may be performed bymultiple devices. Further, while some devices are shown as communicatingwith a network, some such devices may be incorporated, in whole or inpart, as a part of the network.

To the extent the aforementioned embodiments collect, store or employpersonal information provided by individuals, it should be understoodthat such information shall be used in accordance with all applicablelaws concerning protection of personal information. Additionally, thecollection, storage and use of such information may be subject toconsent of the individual to such activity, for example, throughwell-known “opt-in” or “opt-out” processes as may be appropriate for thesituation and type of information. Storage and use of personalinformation may be in an appropriately secure manner reflective of thetype of information, for example, through various encryption andanonymization techniques for particularly sensitive information.

Some implementations described herein may be described in conjunctionwith thresholds. The term “greater than” (or similar terms), as usedherein to describe a relationship of a value to a threshold, may be usedinterchangeably with the term “greater than or equal to” (or similarterms). Similarly, the term “less than” (or similar terms), as usedherein to describe a relationship of a value to a threshold, may be usedinterchangeably with the term “less than or equal to” (or similarterms). As used herein, “exceeding” a threshold (or similar terms) maybe used interchangeably with “being greater than a threshold,” “beinggreater than or equal to a threshold,” “being less than a threshold,”“being less than or equal to a threshold,” or other similar terms,depending on the context in which the threshold is used.

No element, act, or instruction used in the present application shouldbe construed as critical or essential unless explicitly described assuch. An instance of the use of the term “and,” as used herein, does notnecessarily preclude the interpretation that the phrase “and/or” wasintended in that instance. Similarly, an instance of the use of the term“or,” as used herein, does not necessarily preclude the interpretationthat the phrase “and/or” was intended in that instance. Also, as usedherein, the article “a” is intended to include one or more items, andmay be used interchangeably with the phrase “one or more.” Where onlyone item is intended, the terms “one,” “single,” “only,” or similarlanguage is used. Further, the phrase “based on” is intended to mean“based, at least in part, on” unless explicitly stated otherwise.

What is claimed is:
 1. A method comprising: determining a firstplurality of primitives that are within a view frustum of a first renderposition; determining an amount of change associated with each primitiveof the first plurality of primitives; generating a risk map comprising aplurality of probability values that are derived from the amount ofchange associated with the each primitive of the first plurality ofprimitives; and predictively rendering a first set of primitives from asecond plurality of primitives before a second set of primitives fromthe second plurality of primitives based on the first set of primitivesbeing aligned with a first non-sequential set of probability values fromthe risk map that are larger than a second non-sequential set ofprobability values from the risk map that are aligned with the secondset of primitives, wherein the second plurality of primitives are withina view frustum of a second render position that is different than thefirst render position, wherein the first set of primitives have asimilar position in the view frustum of the second render position asthe first non-sequential set of probability values in the risk map, andwherein the second set of primitives have a similar position in the viewfrustum of the second render position as the second non-sequential setof probability values in the risk map.
 2. The method of claim 1 furthercomprising: rendering a first image based on image data associated withthe first plurality of primitives, the first image comprising aplurality of pixels; and associating a probability value to eachparticular pixel of the plurality of pixels based on the amount ofchange associated with one or more primitives from the first pluralityof primitives that provide the image data for the each particular pixel.3. The method of claim 2, wherein generating the risk map comprises:arranging the plurality of probability values in the risk map accordingto the probability value that is associated to the each particular pixelof the plurality of pixels.
 4. The method of claim 1, whereindetermining the amount of change comprises: defining the amount ofchange for a first primitive of the first plurality of primitives basedon an angle by which a surface normal of the first primitive is directedaway from the first render position.
 5. The method of claim 1, whereindetermining the amount of change comprises: projecting a ray from thefirst render position to a first primitive of the first plurality ofprimitives; reflecting the ray off the first primitive according to asurface normal of the first primitive; and defining the amount of changebased on a number of other primitives that the ray reflects off afterreflecting off the first primitive.
 6. The method of claim 1, whereindetermining the amount of change comprises: determining a colordeviation between a first primitive of the first plurality of primitivesand a set of primitives surrounding the first primitive; and definingthe amount of change associated with the first primitive based on thecolor deviation.
 7. The method of claim 1 further comprising:associating a probability value from the risk map to each primitive ofthe second plurality of primitives by aligning the risk map with theview frustum of the second render position.
 8. The method of claim 7,wherein associating the probability value comprises: positioning therisk map over the view frustum of the second render position; projectinga ray through a position of each probability value in the risk map;detecting a first ray that passes through a first probability value andthat intersects a first primitive of the second plurality of primitives;and assigning the first probability value to the first primitive.
 9. Themethod of claim 1, wherein determining the amount of change comprises:determining an amount of color variation between a first primitive ofthe first plurality of primitives and a set of primitives that neighborthe first primitive; and defining the amount of change associated withthe first primitive based on the amount of color variation.
 10. Themethod of claim 1, wherein determining the amount of change comprises:generating a plane based on one or more of a surface normal of a firstprimitive from the first plurality of primitives or a positioning ofprimitives that are adjacent to the first primitive; measuring avariation across the plane; and defining the amount of change associatedwith the first primitive based on the variation.
 11. The method of claim1 further comprising: streaming a first set of pixels that is generatedfrom rendering the first set of primitives to a target device before asecond set of pixels that is generated from rendering the second set ofprimitives.
 12. A rendering system comprising: one or more hardwareprocessors configured to: determine a first plurality of primitives thatare within a view frustum of a first render position; determine anamount of change associated with each primitive of the first pluralityof primitives; generate a risk map comprising a plurality of probabilityvalues that are derived from the amount of change associated with theeach primitive of the first plurality of primitives; and predictivelyrender a first set of primitives from a second plurality of primitivesbefore a second set of primitives from the second plurality ofprimitives based on the first set of primitives being aligned with afirst non-sequential set of probability values from the risk map thatare larger than a second non-sequential set of probability values fromthe risk map that are aligned with the second set of primitives, whereinthe second plurality of primitives are within a view frustum of a secondrender position that is different than the first render position,wherein the first set of primitives have a similar position in the viewfrustum of the second render position as the first non-sequential set ofprobability values in the risk map, and wherein the second set ofprimitives have a similar position in the view frustum of the secondrender position as the second non-sequential set of probability valuesin the risk map.
 13. The rendering system of claim 12, wherein the oneor more hardware processors are further configured to: render a firstimage based on image data associated with the first plurality ofprimitives, the first image comprising a plurality of pixels; andassociate a probability value to each particular pixel of the pluralityof pixels based on the amount of change associated with one or moreprimitives from the first plurality of primitives that provide the imagedata for the each particular pixel.
 14. The rendering system of claim13, wherein generating the risk map comprises: arranging the pluralityof probability values in the risk map according to the probability valuethat is associated to the each particular pixel of the plurality ofpixels.
 15. The rendering system of claim 12, wherein determining theamount of change comprises: defining the amount of change for a firstprimitive of the first plurality of primitives based on an angle bywhich a surface normal of the first primitive is directed away from thefirst render position.
 16. The rendering system of claim 12, whereindetermining the amount of change comprises: projecting a ray from thefirst render position to a first primitive of the first plurality ofprimitives; and reflecting the ray off the first primitive according toa surface normal of the first primitive; defining the amount of changebased on a number of other primitives that the ray reflects off afterreflecting off the first primitive.
 17. The rendering system of claim12, wherein determining the amount of change comprises: determining acolor deviation between a first primitive of the first plurality ofprimitives and a set of primitives surrounding the first primitive; anddefining the amount of change associated with the first primitive basedon the color deviation.
 18. The rendering system of claim 12, whereinthe one or more hardware processors are further configured to: associatea probability value from the risk map to each primitive of the secondplurality of primitives by aligning the risk map with the view frustumof the second render position.
 19. The rendering system of claim 12,wherein the one or more hardware processors are further configured to:stream a first set of pixels that is generated from rendering the firstset of primitives to a target device before a second set of pixels thatis generated from rendering the second set of primitives.
 20. Anon-transitory computer-readable medium storing program instructionsthat, when executed by one or more hardware processors of a renderingsystem, cause the rendering system to perform operations comprising:determine a first plurality of primitives that are within a view frustumof a first render position; determine an amount of change associatedwith each primitive of the first plurality of primitives; generate arisk map comprising a plurality of probability values that are derivedfrom the amount of change associated with the each primitive of thefirst plurality of primitives; and predictively render a first set ofprimitives from a second plurality of primitives before a second set ofprimitives from the second plurality of primitives based on the firstset of primitives being aligned with a first non-sequential set ofprobability values from the risk map that are larger than a secondnon-sequential set of probability values from the risk map that arealigned with the second set of primitives, wherein the second pluralityof primitives are within a view frustum of a second render position thatis different than the first render position, wherein the first set ofprimitives have a similar position in the view frustum of the secondrender position as the first non-sequential set of probability values inthe risk map, and wherein the second set of primitives have a similarposition in the view frustum of the second render position as the secondnon-sequential set of probability values in the risk map.